rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution #149043

aDotInTheVoid · 2025-11-17T23:37:42Z

Historically, it's not been possible to robustly resolve a cross-crate item in rustdoc-json. If you had a Id that wasn't in Crate::index (because it was defined in a different crate), you could only look it up it Crate::paths. But there, you don't get the full information, only an ItemSummary. This tells you the path and the crate_id.

But knowing the crate_id isn't enough to be able to build/find the rustdoc-json output with this item. It's only use is to get a ExternalCrate (via Crate::external_crates). But that only tells you the name (as a string). This isn't enough to uniquely identify a crate, as there could be multiple versions/features ¹ ².

This was originally proposed to be solved via @LukeMathWalker's --orchestrator-id proposal (rust-lang/compiler-team#635). But that requires invasive changes to cargo/rustc. This PR instead implements @Urgau's proposal to re-use the path to a crate's rmeta/rlib as a unique identifer. Callers can use that to determine which package it corresponds to in the language of the build-system above rustc. E.g. for cargo, cargo rustdoc --message-format=json --output-format=json -Zunstable-options).

(Once you've found the right external crate's rustdoc-json output, you still need to resolve the path->id in that crate. But that's """just""" a matter of walking the module tree. We should probably still make that nicer (by, for example, allowing sharing Ids between rustdoc-json document), but that's a future concern)

For some notes from RustWeek 2025, where this was designed, see https://hackmd.io/0jkdguobTnW7nXoGKAxfEQ

CC @obi1kenobi (who wants this for cargo-semver-checks ³), @epage (who's conversations on what and wasn't possible with cargo informed taking this approach to solve this problem)

r? @GuillaumeGomez

TODO:

Docs: Done
Tests: Done

rustbot · 2025-11-17T23:37:46Z

rustdoc-json-types is a public (although nightly-only) API. If possible, consider changing src/librustdoc/json/conversions.rs; otherwise, make sure you bump the FORMAT_VERSION constant.

cc @CraftSpider, @aDotInTheVoid, @Enselic, @obi1kenobi

GuillaumeGomez · 2025-11-18T10:58:41Z

Makes sense to me! Ping me once PR is ready for final review. :)

This is useful for allowing writing run-make tests that test rustdoc-json.

rustbot · 2025-11-19T20:54:14Z

The run-make-support library was changed

cc @jieyouxu

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

aDotInTheVoid · 2025-11-19T20:55:14Z

@GuillaumeGomez ready for review.

GuillaumeGomez · 2025-11-19T21:18:59Z

Looks all good to me, thanks! r=me once CI pass

aDotInTheVoid · 2025-11-19T22:29:34Z

@bors r=GuillaumeGomez rollup

bors · 2025-11-19T22:29:37Z

📌 Commit 457ed4e has been approved by GuillaumeGomez

It is now in the queue for this repository.

…-out-of-spite-does-it-matter, r=GuillaumeGomez rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution Historically, it's not been possible to robustly resolve a cross-crate item in rustdoc-json. If you had a `Id` that wasn't in `Crate::index` (because it was defined in a different crate), you could only look it up it `Crate::paths`. But there, you don't get the full information, only an `ItemSummary`. This tells you the `path` and the `crate_id`. But knowing the `crate_id` isn't enough to be able to build/find the rustdoc-json output with this item. It's only use is to get a `ExternalCrate` (via `Crate::external_crates`). But that only tells you the `name` (as a string). This isn't enough to uniquely identify a crate, as there could be multiple versions/features [^1] [^2]. This was originally proposed to be solved via `@LukeMathWalker's` `--orchestrator-id` proposal (rust-lang/compiler-team#635). But that requires invasive changes to cargo/rustc. This PR instead implements `@Urgau's` proposal to re-use the path to a crate's rmeta/rlib as a unique identifer. Callers can use that to determine which package it corresponds to in the language of the build-system above rustc. E.g. for cargo, `cargo rustdoc --message-format=json --output-format=json -Zunstable-options`). (Once you've found the right external crate's rustdoc-json output, you still need to resolve the path->id in that crate. But that's """just""" a matter of walking the module tree. We should probably still make that nicer (by, for example, allowing sharing `Id`s between rustdoc-json document), but that's a future concern) For some notes from RustWeek 2025, where this was designed, see https://hackmd.io/0jkdguobTnW7nXoGKAxfEQ CC `@obi1kenobi` (who wants this for cargo-semver-checks [^3]), `@epage` (who's conversations on what and wasn't possible with cargo informed taking this approach to solve this problem) r? `@GuillaumeGomez` ## TODO: - [x] Docs: [Done](https://github.com/rust-lang/rust/compare/e4cdd0c24a994fed354081b5f907680a11f2ddc5..457ed4edb184997d5d6f879c6a220bc4d69ff6fd) - [x] Tests: [Done](https://github.com/rust-lang/rust/compare/2e1b954dc52bf7e5a6e9311394df760db37d383f..4d00c1a7ee5e03d1e78801cc01a85dac08ab603b) [^1]: rust-lang/compiler-team#635 (comment) § Problem [^2]: https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/Identifying.20external.20crates.20in.20Rustdoc.20JSON/with/352701211 [^3]: obi1kenobi/cargo-semver-checks#638

Kobzol · 2025-11-20T16:10:51Z

Lemme try something.

@bors try @rust-timer queue profiles=DocJson

rust-timer · 2025-11-20T16:10:54Z

Error occurred while parsing comment: Cannot parse profiles: Invalid profile: DocJson. Valid values are: check, debug, opt, doc, doc-json, clippy

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution

Kobzol · 2025-11-20T16:11:08Z

@rust-timer queue profiles=doc-json

rust-bors · 2025-11-20T18:28:18Z

☀️ Try build successful (CI)
Build commit: 559b9c4 (559b9c48e88d744c5a8625244d6b266158f0f47c, parent: 2c0f4860cc86e2354a55ea04e738848168435c90)

rust-timer · 2025-11-20T18:59:15Z

Finished benchmarking commit (559b9c4): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (primary 0.3%, secondary -0.2%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.7%	[0.5%, 1.2%]	4
Regressions ❌ (secondary)	0.7%	[0.5%, 1.2%]	3
Improvements ✅ (primary)	-0.6%	[-0.6%, -0.5%]	2
Improvements ✅ (secondary)	-0.9%	[-1.3%, -0.7%]	4
All ❌✅ (primary)	0.3%	[-0.6%, 1.2%]	6

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 471.681s -> 472.815s (0.24%)
Artifact size: 388.90 MiB -> 388.80 MiB (-0.02%)

Kobzol · 2025-11-20T19:07:06Z

Looks like this didn't have any perf. effect.

…esolution Historically, it's not been possible to robustly resolve a cross-crate item in rustdoc-json. If you had a `Id` that wasn't in `Crate::index` (because it was defined in a different crate), you could only look it up it `Crate::paths`. But there, you don't get the full information, only an `ItemSummary`. This tells you the `path` and the `crate_id`. But knowing the `crate_id` isn't enough to be able to build/find the rustdoc-json output with this item. It's only use is to get a `ExternalCrate` (via `Crate::external_crates`). But that only tells you the `name` (as a string). This isn't enough to uniquely identify a crate, as there could be multiple versions/features [1] [2]. This was originally proposed to be solved via LukeMathWalker's `--orchestrator-id` proposal (https://www.github.com/rust-lang/compiler-team/issues/635). But that requires invasive changes to cargo/rustc. This PR instead implements Urgau's proposal to re-use the path to a crate's rmeta/rlib as a unique identifer. Callers can use that to determine which package it corresponds to in the language of the build-system above rustc. E.g. for cargo, `cargo rustdoc --message-format=json --output-format=json -Zunstable-options`). (Once you've found the right external crate's rustdoc-json output, you still need to resolve the path->id in that crate. But that's """just""" a matter of walking the module tree. We should probably still make that nicer (by, for example, allowing sharing `Id`s between rustdoc-json document), but that's a future concern) For some notes from RustWeek 2025, where this was designed, see https://hackmd.io/0jkdguobTnW7nXoGKAxfEQ [1]: https://www.github.com/rust-lang/compiler-team/issues/635#issue-1714254865 § Problem [2]: https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/Identifying.20external.20crates.20in.20Rustdoc.20JSON/with/352701211

aDotInTheVoid · 2025-11-20T21:39:09Z

@bors try jobs=dist-various1

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution try-job: dist-various1

rust-bors · 2025-11-20T21:40:02Z

💔 Test for a107de6 failed: CI. Failed job:

Calculate job matrix (web logs, extended logs)

rust-log-analyzer · 2025-11-20T21:40:06Z

A job failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

   Compiling ureq v3.0.8
   Compiling citool v0.1.0 (/home/runner/work/rust/rust/src/ci/citool)
    Finished `dev` profile [unoptimized] target(s) in 22.86s
     Running `target/debug/citool calculate-job-matrix`
Run type: TryJob { job_patterns: Some(["dist-various1"]) }
Error: Failed to calculate job matrix

Caused by:
    Patterns `dist-various1` did not match any auto jobs
##[error]Process completed with exit code 1.

aDotInTheVoid · 2025-11-20T21:46:06Z

@bors try jobs=dist-various-1

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution try-job: dist-various-1

rust-bors · 2025-11-20T23:11:15Z

☀️ Try build successful (CI)
Build commit: be66802 (be668027128df82bcb3a17cf267a40ec00593413, parent: 7281a3bc4b89a010aed05528d648450feb8b69d3)

aDotInTheVoid · 2025-11-21T00:00:50Z

@bors r=GuillaumeGomez rollup

bors · 2025-11-21T00:00:53Z

📌 Commit a47dff8 has been approved by GuillaumeGomez

It is now in the queue for this repository.

…-out-of-spite-does-it-matter, r=GuillaumeGomez rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution Historically, it's not been possible to robustly resolve a cross-crate item in rustdoc-json. If you had a `Id` that wasn't in `Crate::index` (because it was defined in a different crate), you could only look it up it `Crate::paths`. But there, you don't get the full information, only an `ItemSummary`. This tells you the `path` and the `crate_id`. But knowing the `crate_id` isn't enough to be able to build/find the rustdoc-json output with this item. It's only use is to get a `ExternalCrate` (via `Crate::external_crates`). But that only tells you the `name` (as a string). This isn't enough to uniquely identify a crate, as there could be multiple versions/features [^1] [^2]. This was originally proposed to be solved via `@LukeMathWalker's` `--orchestrator-id` proposal (rust-lang/compiler-team#635). But that requires invasive changes to cargo/rustc. This PR instead implements `@Urgau's` proposal to re-use the path to a crate's rmeta/rlib as a unique identifer. Callers can use that to determine which package it corresponds to in the language of the build-system above rustc. E.g. for cargo, `cargo rustdoc --message-format=json --output-format=json -Zunstable-options`). (Once you've found the right external crate's rustdoc-json output, you still need to resolve the path->id in that crate. But that's """just""" a matter of walking the module tree. We should probably still make that nicer (by, for example, allowing sharing `Id`s between rustdoc-json document), but that's a future concern) For some notes from RustWeek 2025, where this was designed, see https://hackmd.io/0jkdguobTnW7nXoGKAxfEQ CC `@obi1kenobi` (who wants this for cargo-semver-checks [^3]), `@epage` (who's conversations on what and wasn't possible with cargo informed taking this approach to solve this problem) r? `@GuillaumeGomez` ## TODO: - [x] Docs: [Done](https://github.com/rust-lang/rust/compare/e4cdd0c24a994fed354081b5f907680a11f2ddc5..457ed4edb184997d5d6f879c6a220bc4d69ff6fd) - [x] Tests: [Done](https://github.com/rust-lang/rust/compare/2e1b954dc52bf7e5a6e9311394df760db37d383f..4d00c1a7ee5e03d1e78801cc01a85dac08ab603b) [^1]: rust-lang/compiler-team#635 (comment) § Problem [^2]: https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/Identifying.20external.20crates.20in.20Rustdoc.20JSON/with/352701211 [^3]: obi1kenobi/cargo-semver-checks#638

rustbot added the A-rustdoc-json Area: Rustdoc JSON backend label Nov 17, 2025

rustbot assigned GuillaumeGomez Nov 17, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Nov 17, 2025

This comment was marked as outdated.

Sign in to view

aDotInTheVoid marked this pull request as draft November 17, 2025 23:48

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 17, 2025

rust-cloud-vms bot force-pushed the is-this-real-is-this-out-of-spite-does-it-matter branch from c7f4e67 to 2e1b954 Compare November 17, 2025 23:55

run-make-support: re-export rustdoc_json_types

8c569c1

This is useful for allowing writing run-make tests that test rustdoc-json.

rust-cloud-vms bot force-pushed the is-this-real-is-this-out-of-spite-does-it-matter branch from 2e1b954 to 4d00c1a Compare November 19, 2025 19:36

rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Nov 19, 2025

rust-cloud-vms bot force-pushed the is-this-real-is-this-out-of-spite-does-it-matter branch 2 times, most recently from e4cdd0c to 457ed4e Compare November 19, 2025 20:53

aDotInTheVoid marked this pull request as ready for review November 19, 2025 20:54

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 19, 2025

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 19, 2025

aDotInTheVoid changed the title ~~rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution [WIPWIPWIP]~~ rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution Nov 19, 2025

GuillaumeGomez approved these changes Nov 19, 2025

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 19, 2025

Zalathar mentioned this pull request Nov 20, 2025

Rollup of 9 pull requests #149119

Closed

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Nov 20, 2025

rust-bors bot added a commit that referenced this pull request Nov 20, 2025

Auto merge of #149043 - aDotInTheVoid:is-this-real-is-this-out-of-spi…

559b9c4

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Nov 20, 2025

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Nov 20, 2025

rust-cloud-vms bot force-pushed the is-this-real-is-this-out-of-spite-does-it-matter branch from 457ed4e to a47dff8 Compare November 20, 2025 19:20

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Nov 20, 2025

Auto merge of #149043 - aDotInTheVoid:is-this-real-is-this-out-of-spi…

a107de6

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution try-job: dist-various1

rust-bors bot added a commit that referenced this pull request Nov 20, 2025

Auto merge of #149043 - aDotInTheVoid:is-this-real-is-this-out-of-spi…

be66802

…te-does-it-matter, r=<try> rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution try-job: dist-various-1

This comment has been minimized.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 21, 2025

matthiaskrgr mentioned this pull request Nov 21, 2025

Rollup of 3 pull requests #149163

Open

rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution #149043

Are you sure you want to change the base?

rustdoc-json: add rlib path to ExternalCrate to enable robust crate resolution #149043

Conversation

aDotInTheVoid commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO:

Footnotes

Uh oh!

rustbot commented Nov 17, 2025

Uh oh!

This comment was marked as outdated.

GuillaumeGomez commented Nov 18, 2025

Uh oh!

rustbot commented Nov 19, 2025

Uh oh!

aDotInTheVoid commented Nov 19, 2025

Uh oh!

GuillaumeGomez commented Nov 19, 2025

Uh oh!

aDotInTheVoid commented Nov 19, 2025

Uh oh!

bors commented Nov 19, 2025

Uh oh!

Kobzol commented Nov 20, 2025

Uh oh!

rust-timer commented Nov 20, 2025

Uh oh!

This comment has been minimized.

Kobzol commented Nov 20, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Nov 20, 2025

Uh oh!

This comment has been minimized.

rust-timer commented Nov 20, 2025

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Uh oh!

Kobzol commented Nov 20, 2025

Uh oh!

aDotInTheVoid commented Nov 20, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Nov 20, 2025

Uh oh!

rust-log-analyzer commented Nov 20, 2025

Uh oh!

aDotInTheVoid commented Nov 20, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Nov 20, 2025

Uh oh!

aDotInTheVoid commented Nov 21, 2025

Uh oh!

bors commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

aDotInTheVoid commented Nov 17, 2025 •

edited

Loading